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ABSTRACT 
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1^ w^ll known assumption of most/o^ the latent trait or item response 
theory model s. curreiTtly being used /e.g., flasch model, three-parameter lo- 
gistic) is that the responses to tWe items on a test are dependent on a uni- 
dimensional latent trait (Lord & mvick, 1968; Whitely & Oawis, 1974).. This 
assumption Dot only holds for this new'Class of test theory models, but also 
holds for many of. the traditional true-score theory based procedures such as 
Kft-20 reliability and item/test^correlation indices of discrimination. For 
s6me applications of the above procedures, thftiassumption of unidimensional ity 
is not a serious problem, since techniques are available for developing uni- 
dimensional sets of items. However, when dichotomously scored items are used, 
as is typically the case for multiple choice items, no widely accepted tech- 
niques are available "for forming a upidimensional space. The problem is fur- 
ther compl icated' by the^ addition of*^ guessing when multiple-choice Items are 
used . . ■ ■ * * ■ 

The technique typically used to sort items into unidimensional- sets is , ^ 
factor analysis, but factor analysis assumes^coritiauaus^jne^s^pesr-atrtM^^ 
with dichotomously ji04:ied^Ttems- resultrin numerous difficulties (Kini & Mueller, 
1978f, enr1"5fcrf?erson' (1975) and Muthen (1978) have developed factor analy- 
sis procedures that are specifically designed far dichotofnously scored i^ems , 
but they can only be used with relatively few Items (25 oc^less) and th'ey do . 
not take jiysssing .into account,. It is important, therefore, that^the currently 
availa.ble procedures for sorting ttems into unidimensional S€ts be evaluated, 
and that'the effects of guessing on these procedures be ^etermlne^d . That is 
precisely the ptirpose of this paper, 



1^ 



Procedures Available fo>r Item Sorting 

In the methodology literature,^ there are several techniques available 
that have the potential to' sort items into unidimensionaV subsets on the basis 
of the responses of^a sample of individuals to the items. These techniques* 
differ in their assumptions and their basic_ underlying model. Along with the 
factor analysis procedures already mentioned, they, Include nonmetriii multi- 
dimnsional scaling and cluster analysis. All so , latent trait theory calibration 
programs in'cQnjunction with goodness of fit tests may be useable to find a 
set that fits a latent trait model.- For the research reported, here', each of 
these technfques will be applied to data-sets of known structure to determine 
which will best recover the structure. However, before describing the re- 
search design in detail, each , technique will be described" to make clear which 
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bf the m$ny variations available 4^ave been used. In most cases ^ the most cotm- ■ 
monly used variation. was selected* ' * ' * 

1 ' ' - ■ 

Factor Analys^is The factor analysis methods most commonVy used are the method 
of principal components and the method of principal factors, These two methods 
differ mainly in the values placed in the main diagonal of the correl ation 
matrix — UO for principal components method* whjich results 1n(an analysis, of 
all of the item v-arlance, and cormnunality estimates fof' principal factors meth- 
'od» which results in an analysis of only the common variance, Both-.of these 
procedures were used to analyze the intercorrelations between the test itenvs 
for this study, _ ! , * 

In 'addition to the particular factor analysis procedure used for the 
study, a correlation coefficient had to be selected for use In determining 
the relationship between the items, Oftentimes phi coefficients are used for 
this purpose* but the magnitude of the phi coefficients are affected by, the dif- 
ficulty of the test Items* sometimes resulting In artifactual difficulty factors 
The common altern ative to the:phi coefiacl-ent in-f-aet^r^ -a:nelVtivwrk"tsrttre"""" 

-iretr^ctroric "correlation. The use of this correlation" al so Jean's t«-some dlf- ' 
f-lculties due mainly J;o the fact that it does not yield a product moment cor- 

.relation matrix, a result the correUtioa matrices produced using tetra- 
choric correlations are sometimes not positivet-semidefinite^ resulting in the 
failure of the factor analysis procedure to yield meaningful results, 

Because of these problems* both coefficients were used in conjunction 
with the factor analysis procedures in this study. In addition, In an attempt 
to compensate for the effects of guessing on the test items,, the four-fold 
tables used to compute the tetracAoric correlations were corrected for guess- 
ing using a procedure develope^f by Carroll (1945), Corrected tetrachoric cor- 
relations were then computed^ based on these revised four-fold tables. Thus* 
the factor , analysis technique^ \*ere applied to correlation matrices developed 
using each of tJhg above three procedures, , - 

Once the basic factor loading matrjces*were obtained using the above pro- 
cedure?* they. were then ^rotated In an attempt to increase the interpretability 
of'the results,; Two coiiinonly used rotation methods were selected: varimax and 
oblimin. The rotations.ifiere performed after the number of factors was, deter- 
mined asi'ng the sqree technique, 

Nonmetric Multidimensional Scaling The purposes of w ul tIdimensjonaV scaling, 
as defined bj; Shepard U972).* are ",_(a) of somehow getting hold of whatever 
pattern or structure may otherwise be hidden in a loatrix of enlpirical da^ta and 
(b) of representing that structure. in a fonn that is much more accessible to 
the hufnan eye,,/'. Since determining the underlying structure in a set of item 
responses will reveal the ynldimensional subsets* this type of^procedure was 
also used in th'is study, - - - * 



The particular v^ipn of nonmetric muUidimensional scaling used for 
thi s 'Study ^a5 the widelyi aval lab! e MDSCAL technique develop'ed by Kruskal 
(1964), This method maps/ the items into a low'dimensional space in such a 
way that'the distances .between thejitemsin that space are* monotoni cair*y re- , 
la ted to val uer af a s imi lari ty coef f ici ent appl i ed to the i tems , Since only 
the ordinal relation of the distances between the similarity coefficients is 
used, many more coefficilents are aRpropri^ate for ,multidimenyonal scaling 
. than aire appropriate fon factor anellysis, ^Because of this Tact, and because 

v8E33:tS?§ '^correlations used with factor analysis have several problems, such as the 
effect of di f ficul ty J ewel on the magnitude of phi coefficients and sample 
size on the stability af the tetrachoric correlation, s,everal other similarity 
coefficients were used/in conjunction with the MDSCAL prdl^£dure, Particularly, 

^ coefficients that werelrobusi to guessing effects were desired. The tdTal 
s'et of , coefficients' used in conjunction with the MDSCAL procedure incVuded; 
phi coefficient, Yul^jS Q, Yule's Y, eta coefficient, a;)proval ^score , Kendall's 
tau b, Goodman/Kruskap.s^gamnia, /and Lijphart index. • , 

/ ' . ' ' ' !_ ^.^ 

Cluster Analysis Cluster analysis is another procedure that has the potential 
for sorting Items in^o' unidimen^ionaV Absets, This procedure also uses sim- 
ilarity coef ficient?*,as an indication of the distance between varia*bles^(i tems 
in this case}. Items that have.high similarity are consider^ed to be ^lose 
together and are, therefoy, clustered together. Those with low similarity, 
are considered far apart and^arl not included in the cl^jstier, 

t 'Two different ciLFStering algorithms were used in thelstudy/ The first, 
called simply CLUSTER, builds, clusters one-at a time, Tt first searches for - 
th^ tw'b most similar items.- These fo:tm the beginning of a cluster'. Next the 
item with the greatest similarity with the items in the claster is found and 
added to the cluster. This procedure 'coritinues until no item has a similarity 
greater than a pre-set cutoff value,- At that point the two most similar items 
noJtj'n the cluster are found to^for^ the^beginning of a new'cluster. The 
clustering procedure continues^as abOve^ ujitil either all Items' are clustered, 
or none can be found above the pre-set criterion level for inclusion, 

' The second clus-tering procedure used, called HlCLbSTER, is a hierarchical 
clustering procedure. In the procedure, the most similar pair of items are - 
clustered and the pair is considered as a new itein. Then the next rrjost simi- 
^' lar pair are chosen and clustered, Ti^is process continues until all items 
are^aired/ Thes^e initial clusters -are, combined to form larger cVusters when 
all points in onfe cluster are paired wtth those in another. Clustering in 
this procedure continues untH;' all of the iteins are ODmbirved into on& large 
cl U'ster , ' ' ' ' , : ' ' ^ 

since neitherof these twcuprocedJres require any special properties of the 
similarity c£)effi-cienis , all of. the coefficients '1 isted ''in MDSC)\L section were 
used in anal yzing' the t^st items, ^ As y/itti tfte MDSCAL procedure , 'i t was hoped 
that ujing coefficients that were not as, depSpd^nt ,on metric information- would^ 
make the procedure"l$ss sensiti-ve to a'li^ss.in^ and difficulty effects , 



Latent Trait Analysis ' AUMugh the application of latent traijt models is not 
usually thought of as an otem' sorting technique, some rproperti'es of the LOGIST i 
program "(Wood > Wingersicy & Lord, 1976) for maximum likelihood estimation of 
the parameters of the three^parameter ■ logistic model suggest, that it might be * 
used to find items that measure ^ .unidimensional trait, Reckase (1979). has 
found'that items with high di^crimirtation parameter estimates from th&' three- 
parameter model tend to be from the same factpr. This is not surprisiog since 
Lord & Novick (1968) have shown t-he latertt trait discrimination pai^ameters ■ tcj 
be related to the loadings of the" test items on the first principal factor of 
a test. Based on these findings jt was .hypothesized that repeated application 
of the LpjSIST program to a test, with low.di scr1m1 n^^i ng itfms deleted afteri 
each stage of the analysis, would yield vun,idimensibnal set of items. There- 
fore, this procedure Was also used , to try^ to fom homogeneous it^m sets. No 
similarity coefficients were required for this procedure. 



Data-?ets Used ih this Study 
' ' . . ~^ - = 

In order to evaluate the^ procedures li^ted'above on the ability to form 
unidimeiisi<jnal item sets» types of data-sets were^ required. PiVst, data-sets 
with known dimensionality and well controlled guessing levels were required 
for an initial evaluation of the procedures. These ^data-sets were produced 
using simulajtion techniques. Although t^hese data-sets dcr^not totaVly match 
re^l data, the fact that they have knownj structure is verjf^ helpful for eval- 
uating the item sorting proc-edures. 

The Second type of data4set required is the actual responses of .individuals 
to test -items, It would be* helpful if somethfng were also known about the 
structure of this data-set^ but accurate knowledge is seldom possible^ Data 
from the Iowa Tests of Educational Develo'SDment , ITED (l^indquist S Feldt, 1972) 
were used to produce this data-set. More detailed descriptions of these data- 
sets are given below. 

Simulated Daia-SetS All of the simulated data-sets used in this study were 
produced u^ing a modification of a method developed by Wherry, Naylor, Wherry 
^ Fallis (1965). This method randomly generates z^-scores to match any desired 
fac*)r pattern matrix using the linear factor. analysis model. For the purpose 
of this study, the z_-scores generated for each 4tem where dichotomized at 
points corresponding to specified proportion correct difficulty indices to 
form 0» 1 scores.' These dichotomous responses were further modified to re- 
flect the effects of guessing." if a wrong response were generated* it was 
changed to a correct response with probability equal to the guessing level 
specified- for the item. 

* J 

■ Data-sets with three different levels of complexity were generated for ' 
this study. These included one-factor* two-factor* and nine-factor data. 
These different levels of complexity were required to'gain understanding of" 
the operation of, the item sorting ^procedures using simple data-sets initi*llly* 
followed by the analysis of more complex situations. 



Guessing level and the distn'butioo of i tern- di f f Kul ty were also varied 
vn producing thefee data*s^ets; ^u^ssing level^s'o/0, .05, ,15, '.25, *35, .45, 
.55, ^65, af1d^*7B were u5ed in the study, Normal and rectangular distri- ( 
bution's of itein>diff1culty were also used. Tne full s^et of data*sets pro- 
duced, for" this study, are giv6n'i,n Table 1, along with a label describing each, 
Data were generated \for 1,000 simulated subjects for all data*sets. 



Insert Table 1 about here 



R6a1 Data-^€^t ■ Only one real d/ta-set was produced Jor this study, This data- 
set was constructed "by 'randomly sampling^33 items fr^m the 69 items on jhe 
E-xpressi^ip subtest of "the JTED" and combining them with a random" sample of 17 
items from the 36 on tfte QMantitative Thinkirvg subtest from the battery, fhese 
tvd subtests were selected aS' bei/^g the most distinct. ■ It was hoped, there* 
fore, that the r*esult1ng 50 -item test would have two factors. The Item sorting 
procedure^ were applied to^the data-set to see if the two 'fac.tors could be 
identifiedt ' \ 

■ , ^ . \ 

' ' ■ * Analyses 

- ^ - * [ 

Two types of^analyses were performed on the s imul atedjand real data-sets 

descrtbed above. First, the faur analysis procedures were'lappl led to th^ 
numerous one factor data-sets to determine the effect of guessing on the pro-^^ 
cedures irt ti\is relatively.pyre case, pata-sets with guessing varying from 
0,5 tt).. .75 were used far this part of the stfudy,, The results of the "appli- 
cation of the Item sorting procedur*es w^re used as a basis for Interpretation 
_pf the further analyse? on the Jiiultidimensional data. sets, 

^ The s":econ"d type of analysi^^ performed was the application of each of the 
Item sprtlTig procedures to the muUldimensional data-sets. In each cas^ the 
"prbceduras .were evaluated on their ability to determine the underlying struct- 
i^re of t+ie (lata y^hen ' guessing was a factor. Both simulated and real data-sets 
were^used I'o.it'hU pa^t of the study. 



Results 



One Factor "O^ta 



The first analysis performed was the application "of the principal co.mpon- 
ents factor analysis techriique to, the one-faotor simulated data-sets. The 
analyS^s'wer'e^perfonn^d on tetrachoric correlations,. A total of nine data- 
?§"ts',^e&pli With €jO Items, a rectangular distribution of traditional^ difficulty 
&nd guessing levels- varying frjDm 0,0 "to .75^ were' used for the ihitial analy- 
ses, 'Al,l*0f the^Jtems in the data-^ets were generate"d using a factor loading 
matrix ^haf^ln?^ .^Silgadings for each .item on the first factor and the remaining, 
variance'^at'tri j^utW to error. " . 



Tfje'ef^^ect of guessing on the^size of thfe* first factor tn the principal 
component analyses of these data-sets U shown in Figure 1, The plot shoves 
the .percent of tfotal test variance accounted for by 'the first factor on the 
test as a function of , the guessing level. The tigur6 also shows the KR-20 
reliability for each test' and the'perceirt of^ variance. accounted "for by t'he 
first ijdctor when da^td-sets generated using normally distributed traditional 
item di fficulties^were used. / ' 



From the figUre, it can be seen that the percent of-^ var^iancfi ^^ccounted 
for by the- first factor of the data.^sets drops off fairly quickly with in-l 
creased guessing, the^effect is somewhat greater for rectingul arl V distri- 
buted difficulty values. -Note thet gues.sing has a much smaller effect on the 
KR-20 statistic. The refeults based on pM coefficient's had" essTentlSl ly the * 
same pattern, but with aj slightly- low^r percentage of variance accounted foi" 
in the first factor. j " 



' Ifrsert F^igdre 1 .{^out Here ' . ■ . 

The effect of guessingf^W the factor pattern of th^ principal component, 
results can be seen in Ta6|0-^2. The table presents the factor loadings for ^ 
■the finst three factors /J^rt unrdtated solution derived from the tetra- 
choric correlations. ^Ij^items in. this test' were arranged from ha^nd.toeasy 
starting with a .01,tj!(ra^^ proportion correct and progressing 'to a .99 

"theoretical proporti^^xorrect at ,02 Intervals 




Insert Table 2' About Here *^ 



The ](5adings on the first factoriShown in Table 2 d.emonstrate two effects 
present m al.l of the other factor analyses. Ftrst, the 'loadings are reduced 
from thf^/heoretical value of ;9 over the entire- range: ^ The loadings for V ■ 
^he easy items/show a. reduction tq the upper ,70*s.. Secondly, the. laadili^s 
for. hard items dramaticarlly show the effects of guessing," As the"diffi> 
culty/of the/Uems increase tlower ,numbered items), the factor /loadings de-;^-*^ 
The decline starts at around item 30, -which, ha^ a theoretical per-,^-.^ 
'ce'rlTcorrect of .59, ' • ^ ' V 

( The other two factors shown in Table Z .seem to bft two guessing factors. . 
Factor II has moderate positive loadings for the ^moderately difNcJtilt it^s, 
while Factor H fjas moderately positive loadi^ngs fo'r some of the yveVy "tllff.i- 
cult items. It seems that these two factors were required to^ a^ccoynt/^^ifr^the 
curvtlinear nature of the guessing effect; The phi coefficient *;.^^yfts' were 
similar, but with lower f^ttor loadings. Also, phi coefficients ^i^su^lted io 
dlffic(j,lty"factors defined by the very easy and very dlfficuVt'Xt^ms. 
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The factor analysis procedure was also^run on the tetrachon'c correlations 
obtained using CarrolT's (1945) correction for guessing. The first three 
factors of the factor^ pattern mati>ix^from the principal "component analysis ' 
are presented Vn Table 3\ As can'be'seen^ from^this table, t)ie first factors ^ 
are rfow closer in magnityde \o the^ .9 used to generate' the data. The guess- 
ing'effedt for the difficult items has also deminished- Further, the third 
.factor no longer seems to be related to guessing. Thus, the correction for 
guessing 4oes seem to have sortie effect ^ but the pres€nc& of the second factor 
that still seems to be related to guessing Indicates that al'l of the gu-essing 
component has not been removed* . " ^ ,^ ^ 



Insert Table 3 Atx)ut Here 



The MDSCAL and cluster analysis procedures gave conceptually sinlllar re- 
sults to that obtained by the factor analysis when applied to th.e one-factor 
data with a guessing component, but of course the representation was ^di f ferent 
due^ to the different fom* of the artalysis niadel / Thfe effect of guessing on . ^ 
/the HDSCAL results is shown'^by the dispersion -af the hard items fnim the 
■tight grouping of the easier iteftis on th€ test. Fi.gure 2 shijws this effect 
for a 50 it-em simuUted test with rectangular distribution of item difff- 
^ cutties and ,25 guessing level. Tlie MDSCAL analysis was performed on tetra-- 
choric correlations, Figure^S shows the same pTot,wtieri no guessi/ig effect 
was present'jn thfe data/ For both plots, the items were numbered from haErd 
to .easy with the low numbered items being the harcjf items. 

• ^ • . . .' ' ■ ■ I ■ • 



Insert Flggres 2 & 3vAbout Here 



The CLUSTER and HICLUSTER procedures resulted in sijnilar guessing effects 
to those shown for the MDSCAL procedure. The 32 easiest items (with one or 
two ejcceptionsj were grouped- together into one major cluster, while thfe dif- 
ficult ttems that were' af fecte^d by guessing were included in several smaller - 
clusters, These smaller clusters were absorbed into the large cluster when 
no guessing CDrriponent was present. All of these results for the other simi- 
larity coefficients 'werfe muct) the same so *they have not been presented bere 
for the sake of brevity, ^ - ^ ^ 

The appll^tion of ihe L06IST program to the- one factbr dat^with a guess- 
ing component gave very good results. The program gave uniformly high esti- 
mates for the discrimination parameter?* evenly spread difficulty parameter 
estimates, and accurate estimates of the guessing ,1 evel , This fact was ^c^t * 
surprising since the data /net all of the assomptipns .of 'the three-parameter^ 
1 ogistit model. Since iguessing Is built into this model, guessing effects 
are of no concern. The prOgram estimates the guessing level rather than 
having the results ijjstorted by a .guessing effect. 
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Two Ractor^td ' . * . . ■ - ^ 

"The. real test of a procedure designed to sort items into unidi*mensio"nal 
sets is whether the, procedure can, recover the known structure of a set of 
data. Th^ simplest data-set that can be used to^^evaluate the- procedures is 
one that is combosed of two factors of rougWy equal slife. Three data-sets 
of tfris type w^ire used for the initial evaluation-of the procedures used in 
this study (Seeipable .1). All three Kere generated using a factor loading 
matrix that had:an iteln factor load1nglof \9 on one factor ancJ a 0,0 loading 
on the other. Hal-f of^the items loaded on the first factor and half on the 
second. Rectangul ar" and normal distrioutions of guessing were "used*^. The 
guessing levels used were*.0, \Z, and, .25. 

Under the no guessing concjition, all of the procedures except the CLUSTER 
procedure using eta coefficients did a good job of sorting the items into 
groups b^sed on 'their membership on the two factors, In^the case of the 
CtUSTER procejSure using eta coefficients, many small clusters werS formed in- 
stead'of two large ones. A similar result was. obtained when the procedures 
were applied to the two fact<^flata-set with normal distribution of diffi- 
culties and guessing .component, and with an average guessing level of .2^ 
All pf the procedures except the CLUSTER^ procedure^ us ing et^a coefficients 
could easily sort the items .into uni'dimehsional sets, j ^ , ■ 

When the two-factor data with rectanguUr distribution of item diffi- 
culties and guessing constant at -25 y^s used', <juite variable results wece ob- 
tained. The principal component and principal factor results on tetrachoric 
correlations were able' to separate the two sets of items, but the^reSults were 
not nearly as clear as foj:.,the' previous two data-sets, 

.Figure"4 presents a (riot of the items from the two factors in the space 
defined by two VARIHAX f^^tor^ from the principal compjonent analyses of the 
tetrachoric corVel atio^is . As can tfe seen from the"plo|^ the items^ from the 
two factors are arranged along the two axat of-ythe ^solut101 , The more diffi- 
cult ftems with a large guessing componTnt are closer to the origiq. The " 
closeness of the two sets of items in the area around the origin could indi- 
cate poss-ible'problems in item^class/fication under less optimal circumstances, 



Insert Figure* 4 About Here 



The results of. the HOSCAL analysis of this data varied in quality depend- 
ing on the similarity coefficient used^in the analysis of. the data\ The con- 
figurations based on the. eta , coefficient , approval score., "agreement coefficient, 
and Lijphart -index resulted in poor discrlwf nation of the two factors present 
in the data,^ the, configurations based on the Yule's Y, Yule's Q, Goodman/Kruskal 
gamma, phi coefficient, Kendall's tau b, -tetrachoric correlation and the tetra- 
choric correlation corrected*- for guessing were easily divided into the sets of 
items for each factor. - Figure' 5 presents a, plot of the MOSCAL results for the 
two dimensional solution l)ased on the tetrachoric correlations corrected for 
guess ing . - ^ 



Insert Figure 5 About Here 



The cluster analysis procedures* did not do nearly as^good a 30b of sort 
ing the items 1 into. the two sets as did the MDSCAL and factor analysis prbced 
ures. lh% major problem- in using the cluster analysis procedures was in de- 
termining the number of clusters to use as a result, When two clusters were 
specified for thra HICLUSTER procedure the approval sdoY^e correctly classified 
most of the items!, From^ eight, to/22 item's were misclassified using the other 
-eofrff4c>ents witri ph4 coefficients .^fving poorest restHts^, it is .interesting, 
that one of the v'orst "coefficients foK use with the MDSCAL procedures, the* 
4PE>roval score, i as the best for use with the cluster analysis procedure, ^ 
The results of trfe CLUSTER procedure were tnuch worse, with seven to 37 items 
misclassified, Many of the misclassif ication errors were for tne\mbre diffi- 
cul t vt^lfis . - ' ' ' I ' 

Tneappl ication of the LOGIST procedure to the two factor fata-sets -re* 
suited in vgcy* good results regardless of the dif f iculty distri( ution and the 
level of guessing. In all cases, the ^1tems from "one factor had high discrlni- 
ination parameter estimates wnile^those froirl'the otrt?r had low oarameter esti- 
[mates\ Tnere was no" difficulty in sor/tin^ the items into tl&e tvfio d^ta-sets, 

J > ! ' " 

Based on tne rather stringent eval uatlon of thje procejJures iusing. the two 
factor data-set with rectangularly distr\1buted item! diff icuUiea^and ,25 guess 
ing level, it would seem that the factor analysis, nonmetric mu.ltidimensional 
6caling^ ano latent traWt analysis have promise foK sorting ftems into u/ii 
ojmensional sets. Cluster analysis ,does not look promising 
ficulty in determining the appropriate number of clusters, 
classes of prbceckjres will now be further evaluated using a 
set. I 



because of tne dff- 
the thpee selected 
nine fJictoV data- 



l^ine Factor Pata . . 

In order to evajuatg two procedures using s,imulated data-set that had more 
realistic properties tnan the two-factor dati-sets, a nine-fact<yr dataVset was 
produced, Tilis data-set was. generated to l^ave a general first factor^w^it^h ;5 
loadings for each item and eight other factors with *5 loading^r for five or, 
six .items and near zero loadings for all other^, Item difficulties and sues s- 
in^ 'levels were normarlly distributed for thj??,, data-set, The mean guessing ■ 
level was set at .2, The test data was generated us'lng positive loadings -on 
the factors for all items since it was felt that the items on the typicar\ 
ability test were positively correlated. 



^The principal component analysis of the nine factor data using tetrachoriC 
ct^rr&lations resulted in a very clear*s6rting of the item sets p^esent in. 
tne data. After the ,VARIMAX rotation was performed on the 12 fa :tor princi- 
pal- component solution, each of th> sets of five or six items wai^welJ defined ' 
by hign loadings (,6(J to -.lo) on its cfWn factor. All other loadings orr the 
factor v<ere J)elow .20. The analysis of phi coefficients- gavfi. ^^iiniUr re^ __ _ 
suit, b.ut w1tti lower overall magnitude of the factor loadings, Since a nor- 
mal distribution of Uen) difficulties was used for this data-sets, eliminating 
>tne items, of extreme dif f iculty^,_ no difficulty factors seemed to.be present in 
the results. . " " " . ' ' - ^ 

iJosi.o.f_the^pplicAidojis^of„theJJ1DSCAL procedjjre also did a good job of 

'sorting the it§nis into unidimehsional'sets. /As was TowTd prei^ioiis^ly^ with the 
two,*dimensiona"i data, analyses-based on Yule^s Q; Yule!s*Y, Goodman/Kruskal's 
garrma,^ Kencla\rs tail b, phi coefficient, .tetrachoric correlation and corrected 
tetrachoric correlatioh"'perf onned well. Those based orxjtne 49^^^^>i^nt score, 
i;he approval index, Lijphart's in"3gx and the eta coefficient did ^ot adequately 
sort the items, figure ,6 shows the results for YuleU.Q indSx as an'-example ' 
of these resul ts. ' * , / ; , 



Insert Figure S^jhere ' 



Appilicatipn of the LOGIST procedure to the nine factor data also resulted 
In an accurate' sorting of the items into the appropriate item s^s, but th? 
procedure was lon^ and expensive vn terms of ^computer ^tnme* "The initial calin 
Dration of the items yielded discrimination parame-ter estimates for all ''items 
in tne', 4 to ;S r^inge. By deleting the items ^with the lowest discrimination 
parameter ^estimates. and recal ibratihg > a unidimensional subset bf items was , 
eventually obtained* ^ Unfortunately, 10 iterations of item deletions and re- 
calibrations were requfred to get one unidimensional set.. To totally'sort , ^ 
tne items into sets, required* a prohibitive amount of computation. Therefore 
It "seems that the LOGIST program 4o^s OQ^t off-er a viaWe item sorting, pro- 
cedure when numerous dijiensibns ara 'Suspected to be in a test. LOGIST did 
accurately estimates the guessing level of the items, h'(Mever, a fact that may 
prove useful in other situations, ^ - . 

. iT ^.Qata ' ' , . . \'. , * ' 

*The analysis of the two fact?)r and 'nine factor^data indicates that the^ 
factor analysis and* nonmetricmultidimensional scaling "pro.oedures" usihg tne 
appropriate statistics did a good job of sorting the 'dichotomous,ly scored test- 
itfims into AjndimensionarV sets. UnfortiAiately , simulatfid 'data-sets never have ^ 
the same properties a'S real data, so the results given above are only tenta- 
tive. Ti^e procedures. must be evaluated using real data, to ^i9% a true indi- 
cation 'of. their jworth. - ^ , ' ^* ' 

As/descnioed ^earlier, a real data-set composed of verbal expression and 
quantitative items fi^om^the ITED was produced from the fifll test battery for 



tne purpose of performing a realistic ev^aluatitin" of the procedures. The data- 
set produced was composed of 33 Expression items and 17 Quantitative iteins. 
Tnese/two suDtests ^ere.combined to forn) !i reasjonable two-fa<;tdr data-set using 
real data. , " : . \ ' - ^ . " ■ ' 

Tne first analysis performed on this data-set was fajctor analysis ^af the 
inter-item phi » tetrachoric" and~corrected tetrachorJc correlations/ The re 



5^ulrs of ^e principal component and principal factor <, 
were approximately the same, so only the VARIMAX rgtat: 
^ponent results wjlUbe discussed* the ^^esults of this 



good. Twenty- t^^o of jthe 33- Express ion iteirtS were correctly classified^ while 



14 0f l7 Qudnti tatlve items" were classified together, 
were correctly placed into the content categories. 



to 



The MDSCAL procedure dicf not. fair nearly as well 
unidimensional item s^ts. In no case could a Qlear 



nalyses of the d^ta a 
on of the principal com- 
analysis were fairly 



Thus 72% of the it6ms 



n sorting ^he items in- 
distinction h€ made be- 



tween the conteht" areas Regardless of the coefficient heirtg analyzed. The 
procedure did tend to cluster together items frorti the same..content area» but 
tjiese clusters/could Inot be identified without knowledge of ttie ditnensions ,\ , 
tiemsielves, ^ Figure ^.sl)ows the results Of the MDSCALj procedure for the ITEO 
dhta using Yule's Q Statistic* As can be seen» the cjuantitative itens cluster 
tjgetJier^ but it woaVd be dif^icuH to separate the dontent arets without ^ome 
aavanceq knowledge pf the cbnterrt breakdown* 



Insert Figure 7 about here ' 



V 



.Discussion and Conclusion 



Tne purpose of this paper has been to examtne the capabilities of the 
various procedures for sorting items into unidimensional subsets* the pro- 
cedures^ Inplude: factor analysis^ nonmetric multidimensional scaling^ cluster 
analysi s and latent trait analysis / Both 'simulated and real data-sets of 
Krtown structure were used to evaluate/ the procedures. 

The analysis of the one-factor dat;a-sets with varied levels of guessing- 
snowed tn^ detrimental effects guessing could have on these techniques. Guess- 
ing reduced the magnitude of the factoids in'the data and aometimes induced 
otner^ "guessing'; factors ,no;t related to ability or itenv^ntenti However, 
tnesedetrimental effects were not noticeable when ^^P^^ ff iculty and guess - 
tng were normally/distributed. Items of more extrefifS^f^icul ty were required 
before guessing had a serious effect: . ^ 

^Application of the procedures to the two^^nd r>ine-factor^data-sets showed 
tftat factor analysis and multidimensional, sq^nng ^had promise as techniques ^ 
for sorting items into unidimensional sets. .Cluster analysis performed well " 
in some ,cases,^but was dropped frpni'^cons idejjation because no good way was 
known to deter'mine the number, of clusters.^ Latent trait analysis also per- 
. formed weVl in some.cases, but/the computational requirements of the procedure 
made. it an intpra^t-ical procedure for general use. 
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^ "The analysis of the real test data showed that the factor analysis pro--' 
' cedure was the-only one which cocild do a reasonably good Job of soY^ting i, terns 
Into um'dlmension'al sets under., realistic conditions. Although the MOSCAL 
pi aced 'items from the same content area near each. other in the space defined 
Dy tha proce<(ure^ it ^1ji not adequately differentiate the items in such a way, 
tjhat they^, cQul^^De accijratelx classified- Thus/ to -^e extentfthat the re- 
sults. of this study'can be generalized beyond the sim^ated. and real data- ^ 
' sets" used, .factor analysji .st^ll seems to be the method most capable" of sort- 
ing items JntounidimensipnaV sets / ^ ' ^ % . \ . ^ ' 
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Table 1 



List of Simulated^Data-tSets 
-If* r '' 



Dimensionality of Data Set 



LabeV 



. 1-Factbr 



SD150N, 
'SD150R-, 
SD150R, 
SD150N, 
SD.150R, 
SDISON, 
SD150R, 



CG15» 

CG35 y 
CG45; 
CG65, 
CG75 



2-Factor 
9- Factor 



SD150R,GG0p» SD;5QR.CG05 
SD150R.NGlOD150fl.CG15 
50150R.N^5^ SD150tkCG£5^ 
SD1'5QR'. CG'SS^^^tSCiN . CG45. 
SD'l5jDN.CG55, Sl)150R.dG55 
$D150R.CG65, SD150N.CG7§ 



• SD250R.CG00» SD250N.NG20, SD250R.CG25 



SD950N.NG20 



,*The label of the* data set describes , the (fatarset. 'the- first two letters, 
stand for simulation data'. 'The next three or four dfglts' tell/the number 
of^ifadtors the'number of Iterns. All data-sets CMtaIn' 50 items.' The 
letter foHoviin^.the, 50. tells the dis'trlbutllan of traditlpnal. Item diffi- 
culties: M drR meaning, nonjial or retangular^ respectively, following the,_ 
period. isjCfi or NG standing for 'constant or normally distributed gues'siftg. 
The final^o digits give'the guessing level. The values given are the' 
guessing level'for.CG dat?-sets or the mejan guess4ng level for NG data-sets, 



^ . ' 'y- • . . . Table 2 , ;• \ ■ - 

/Factor Pattern Matrix for a Threg-'facW Principal Component 

.Solution to Data-Set SD150R .'CtS25 ■ 
'I'temr • ' , f . Factor 



TTT 



" 1 , . - ' 06 13" 31 

. 09 ■ , ' -23 ^ ■ 07 

3 I . . * ^ 17 15- 36 

4' . V*- > • . 12\ , 29 42 

' 5 : •■ V • . W. ' 36 ■ ■- 31 

3 25 25'^ ' . 19 

■-7' . 3.0. . 27- • 38 

8 ■ ■ ' 29 '43 ■ 05 

9 . ■ 41, 23 , 37 
;0 • ■< ^39 ' 31 ■ , 32 
.1*1 . 53 . 25' . -14 

12 . ' . , '49 ' • 33 18 

13 - ■ - y , . • 47 - " ' ^ ,36 -14 

14 - ' , ' ,54 ■ • ■ 32 . ■ . '14 
'15v ■ ■ ^ - . ■ ■ • ■ .-55. ^' ■ ■ , 40 •■ ' -03 
tS' - - - -6^. ' 31 ' -01 
17 ' ' - '.„^^ • " : 36 ' -04 
18' ■ ■ ^ ^6- ' .24 ' ■ -15 
19 . : '62 • ■ V. 32 ■ • > -05 



.20 . ■ . • '' 66' ~t:-^ ■, 28 . - -07 

21 ■ • . ■ ■ ' ■ ' 63 • ' ' 34 . •■ ■ -26 

22; ■. * ■ ; "68 '.-^ 26 ■ ' - - ■ 01 

'23. ■ 66 ■. 28 -11 



^- 24- • ' U -T^. 19 -07. 

. 25 ■ * „ .-. 71' " : 20' • ■ . -19 

26:^" ^ ■ • ■ -.72 " . ■ . , 18 - . . -14' 

' 27 '■ 'r - \. -^Z 70. .• ■■ 08 — 06 ^, 

. 28 . ■ /. - ■ 76 . ■ '10 '• -11 

. 29 ■ ' - '73'- ' . ■ 14 - 6 . -13. 

-30 ■ ] ■ 75^ '■■ 09 . -14 . 

' 33* J ^ ■. . 78 -10 . ■ l7 ' 

. 32; - , r 7? ■ . . * ■ '06, (-21 

33 ... . 77 '-09 - ■ . ■ ' -07 

34 • . ■ , ' ■ 79' -J}8 ■ ■ ' . ^09 

' 35*. ■ *■ 75 ' * ■ -09 ■ - , ^ --15 1 

. 36 " . • ^: ■ 81 . -18 ' . ■ ■ • . -01 " 

.37 ■ ■ 77 : • -08 . \ -07 f.. 

•38 , " " . 78' -14- . * ■ -10 !■ 

.39 , , ■ 78 . . ." . -20 • « ■ -13 i' 

*40 . -a 78 ■ , -25 , • -05 1 

'41 . , ? r .77 . . . -23^ rOl ' 

* 42 . '■ , 77/ . '-23^ . . -05 ; . 

■43 ■ ' . .79 ■ - -32 . -01 

44 ■ , ' ■ 79 . -35 . 13. 

■ 45" -.- , 77 ■. ' - . -39 01 ■ . 

^46 : V • '77^ -36. ' ■ ^04 ■ 

47 T ■ ■ 76^'.'- , -37 ■. . 19'- 

48 . ^' • v ■ 79"- • , -4? ^ ^ ■ . 17 

49 ■ . ' 75u * • • '1 r> -43 ■ . » 02 
.■50 ■ y. *v * 63V ^0-60 . ,.55 ■ 

— ' — : : '■ ; [ r-r- — 

Note: • Va^lues aWfttjesei^ted without decimal points for'an un'rotated solution 
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" ' * FatWr pattern* Matrix, fdf a'Jhre'e-Factor Principal Component * 
* . Solution for Oat&vS&t SDlSOR.CGZS Corrected for Gu-essing 
Item • • : ' - ■- ; . Factor - ^ 

; ■ ' - . ..y : • 1*' — rr-; m — ■ 



-er|c ^ 



— *-T= — 


i 






1 ' 


. 64 .. 


75 


117 . , ^ 
-35 . - 




•■' . ' 78 ■ / 


,92 


1 i ■■ . 


99 


48 


'(-02 


66' ■ '• ' 


' 63 


-•42 


^ * * 


. ■ 71 ' • 


59 . 


05 


6 , . . 


' : '75 ■ "■ 


3D . ■ 


-16 


7 


'.91 


14 


21 


8, . 


T 64 ■ 


51 


' . -51 


9" 


90 ' ( 


09 


19 


ro 


.72 


33 ' . ^ 


" .19 




100 


■ -05- 


' Ol 


12 


95. " i 


-06 


' 45 


' 13 ^ \ 


► 79 . 


-01 


'-33 - 


14 


95 ' ' 


12 


^2 




. ■ 94 


10 


^-22 \ 


15 


'100 


00 - 




17* 


80 


■ 17 > 


P9 


18 


98 


. -14.' 


09 . , 




87 


•19. - . 

* 


. 05 


20 • ^ 


94' . 


■ 13 • 




21 


. 82 


13 


■04 


22 


93 


2a 


00. 


23 


86 


19- ♦ 


-25 ' 


'24 . \ 


^ 97 


- 03 ■ 


V 12 . 


25 " 


93 ■ 


' 08" , 


^01 ^ 


25 > ' - ' \ 
27'. ^ 


■90 - •■ 


---01 " ■ 


18 


■ 89- 


06 


15 


28 ■ . 


95 ■ . ' ' 


00 ' . • . 


11 


^29 


91 


12 ' 


-02 


^ 30 ' 


91 


02 


-3.7 • , 


31 * ^ 

32 ■ ■ - ' 


92 


-08 V '1 


-19 - - 


84 , 


-14 


-51 . 


33 ' 


'•88 


I2I ' ^ 


36 


34 ■ 


91 


* -17 


07- / 


3& . - 


82 


■ r43 


19 


36 


' 90 . 


-48 


' -18 


37 * ^ 


92 


' 14 


' ' 06 * 


38'-- ' 


•87 * 


.-22- 


- ■'-04'.- 


39 ' ■ ■ . ■ . 


■ 88 


' > -17 


-12 


40 


84 


-54 


- 12 


41 - , 


86 


-13 


. -12 


42 




• -23 


. -15 


43 


87 - 


-46 


-13 


44 " - 


93 


-14 . - 


29 


45 


86 'A 


-24 ' , 


16 


46 ' 


88 , 
94 


-29 


-36 


.47 


•^3 


16 


48 


95 • • 


-43 • 


02 . . 


49 




-71 . 


' -13' 


' 50 ■ 




-52 




Note: Factor loadings ar^e presented without- decimal pointsi ' , ^ . 
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